difference approximation
LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation
Kim, Eunsu, Suk, Juyoung, Kim, Seungone, Muennighoff, Niklas, Kim, Dongkwan, Oh, Alice
We introduce LLM-as-an-Interviewer, a novel paradigm for evaluating large language models (LLMs). This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the MATH and DepthQA tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM's strengths and weaknesses. This report offers a detailed snapshot of the model's real-world applicability. The code for our framework is publicly available at https://github.com/interview-eval/.
Data Mapping for Restricted Boltzmann Machine
R estricted Boltzmann machine (RBM) is two - layer neural nets constructed as a probabilistic model and i t s training is to maximiz e a product of probabilities by the contrastive divergence (CD) scheme . In this paper a data mapping is used to describe the relationship between visible and hidden layer s and the training is to minimize a squared error of the reconstructed visible layer by the gradient descent or a finite difference approximation . T his paper presents three new findings: 1) nodes on visible and hidden layers can take real - valued matrix dat a without a probabilistic interpretation; 2) the famous CD1 is a finite difference approximation of gradient descent after ignoring the second - order error; 3) activation can take non - sigmoid function s such as identity, relu and softsign. The data mapping p rovides a unified framework on dimensionality reduction, feature extraction and data representation pioneered and developed by Hinton and his colleagues . As an approximation of gradient descent, the finite difference learning is applicable to both directed and undirected graphs. N umerical results are performed to confirm these new findings on very low dimensionality reduction, matrix data and flexible activation s . Keywords: Restricted Boltzmann machine, data mapping, squared error, contrastive divergence, gradient descent and finite difference .